Systems and methods for optimizing a machine learning-informed automated decisioning workflow in a machine learning task-oriented digital threat mitigation platform

ABSTRACT

A system and method for adapting an errant automated decisioning workflow includes reconfiguring digital abuse or digital fraud logic parameters associated with automated decisioning routes of an automated decisioning workflow in response to identifying an anomalous drift or an anomalous shift in efficacy metrics of the automated decisioning workflow, wherein the automated decisioning workflow includes a plurality of distinct automated decisioning routes that, when applied in a digital threat evaluation of data associated with a target digital event, automatically compute a decision for disposing the target digital event based on a probability digital fraud; simulating, by computers, a performance of the automated decisioning routes in a reconfigured state based on inputs of historical digital event data; calculating simulation metrics based on simulation output data of the simulation; and promoting to an in-production state the automated decisioning workflow having the automated decisioning routes in the reconfigured state.

TECHNICAL FIELD

This invention relates generally to the digital fraud and abuse field,and more specifically to a new and useful system and method fordetecting digital fraud or digital abuse and evolving underlying machinelearning models in the digital fraud and abuse field.

BACKGROUND

The modern web and Internet enable entities to engage and perform anincalculable number of activities. Many of these activities involveuser-to-user activities, user-to-business activities (or the reverse),and the like. These activities between users and between users andorganizational entities over the web often involve the access, use,and/or exchange of information by one or more of the parties of theactivities. Because of the malleable nature of the digital realm thatthese activities operate within, there arise a countless number ofdigital threats by digital actors that aim to commit digital fraudand/or digital abuse using online services and/or Internet-accessibleapplications (e.g., web or mobile applications). Additionally, some ofthese bad digital actors may also aim to misappropriate the information(e.g., hack) being exchanged between legitimate entities to theseactivities. These digital threats may also be perpetrated by maliciousthird parties who seek to unlawfully or otherwise, impermissibly takeadvantage of the data or information that is exchanged or, if notexchanged, data or information about the activities or actions of usersand/or businesses on the web.

Other digital threats involving a malicious party or a bad digital actorthat acts unilaterally (or in concert with other malicious actors) toabuse digital resources of a service provider to perpetrate fraud orother unlawful activities that are also of significant concern tolegitimate service providers and users of the Internet.

While there may currently exist some technologies that attempt to detectdigital fraud and digital abuse or other malicious digital activitiesover the Internet, these existing technology implementations may notsufficiently detect malicious digital activities over the Internet withaccuracy and in real-time to provide an opportunity for an appropriateresponse by an affected party. Additionally, these existing technologyimplementations lack the capabilities to detect new and/or never beenencountered before digital threats and automatically (or nearautomatically) evolve the technology implementation to effectivelyrespond and neutralize the digital threats.

Therefore, there is a need in the digital fraud and abuse field for adigital fraud and abuse solution that enables effective detection ofmultiple and specific digital threats involving digital fraud and/ordigital abuse via digital resources of a service provider. Theembodiments of the present application described herein providetechnical solutions that address, at least, the need described above.

BRIEF SUMMARY OF THE INVENTION(S)

In one embodiment, a method for adapting an errant automated decisioningworkflow for improving digital fraud or digital abuse mitigationincludes generating a succeeding automated decisioning workflow byreconfiguring an incumbent automated decisioning workflow based ondetecting an anomaly in automated decisioning outputs of the incumbentautomated decisioning workflow, wherein the incumbent automateddecisioning workflow comprises a plurality of distinct automateddecisioning routes that, when applied in a digital threat evaluation ofdata associated with a target digital event, automatically compute adecision for disposing the target digital event based on a probabilitydigital fraud or digital abuse associated with the target digital event,and wherein generating the succeeding automated decisioning workflowincludes: tuning at least one automated decisioning route of theplurality of distinct decisioning routes of the incumbent automateddecisioning workflow based on one or more in-production metrics of theautomated decisioning outputs of the incumbent automated decisioningworkflow; evaluating the succeeding automated decisioning workflow,wherein the evaluating includes implementing a route simulation of thesucceeding automated decisioning workflow that simulates a performanceof the at least one automated decisioning route of the plurality ofdistinct automated decisioning routes; computing one or more simulationmetrics based on simulation output data of the route simulation of thesucceeding automated decisioning workflow; and replacing the incumbentautomated decisioning with the succeeding automated decisioning workflowif the one or more simulation metrics satisfy or exceed one or moreefficacy benchmarks.

In one embodiment, the tuning the at least one automated decisioningroute includes: estimating a rectifying edit that amends one or moredecisioning criteria of the at least one automated decisioning routebased on attributes associated with the anomaly in the automateddecisioning outputs of the incumbent automated decisioning workflow.

In one embodiment, one of the one or more decisioning criteria of the atleast one automated decisioning route comprises a machine learning-basedscore threshold, and the rectifying edit includes adapting the machinelearning-based score threshold by increasing or by decreasing a machinelearning score value or a machine learning score range of valuesassociated with the machine learning-based score threshold.

In one embodiment, one of the one or more decisioning criteria of the atleast one automated decisioning route comprises one or more adversefeature conditions, and the rectifying edit includes adapting the one ormore adverse feature conditions by deleting one of the one or moreadverse feature conditions or by augmenting the one or more adversefeature conditions with a new adverse feature condition.

In one embodiment, one of the one or more decisioning criteria of the atleast one automated decisioning route comprises a machine learning-basedscore threshold defined by either a machine learning score value or amachine learning score range of values, and the rectifying edit includesadapting the at least one automated decisioning route to include one ormore new adverse feature conditions.

In one embodiment, each of the plurality of distinct automateddecisioning routes is arranged in a predetermined order within asequence of the plurality of distinct automated decisioning routes, andthe tuning the at least one automated decisioning route includesre-positioning the at least one automated decisioning route to a newposition within the sequence of the plurality of distinct automateddecisioning routes.

In one embodiment, each of the plurality of distinct automateddecisioning routes is arranged in a predetermined order within asequence of the plurality of distinct automated decisioning routes, andimplementing the route simulation of the succeeding automateddecisioning workflow includes: simulating the performance of the atleast one automated decisioning route independently and outside of thesequence of the plurality of distinct automated decisioning routes.

In one embodiment, each of the plurality of distinct automateddecisioning routes is arranged in a predetermined order within asequence of the plurality of distinct automated decisioning routes, andimplementing the route simulation of the succeeding automateddecisioning workflow includes: simulating the performance of the atleast one automated decisioning route in-place and within the sequenceof the plurality of distinct automated decisioning routes.

In one embodiment, implementing the route simulation of the succeedingautomated decisioning workflow includes: defining a simulation corpus ofdata comprising historical digital event data input to the incumbentautomated decisioning workflow during a historical period; andsimulating the performance of the at least one automated decisioningroute based on inputs of the simulation corpus.

In one embodiment, implementing the route simulation of the succeedingautomated decisioning workflow includes: setting the route simulation toa shadow mode, wherein during the shadow mode a copy of live digitalevent data passing through the incumbent automated decisioning workflowis additionally passed through the succeeding automated decisioningworkflow, wherein in the shadow mode automated decisioning outputs ofthe succeeding automated decisioning workflow are not exposed via anapplication programming interface.

In one embodiment, during an implementation of the shadow mode, theroute simulation is performed for all automated decisioning routes ofthe plurality of distinct automated decisioning routes of the succeedingautomated decisioning workflow.

In one embodiment, the computing the one or more efficacy metrics basedon the route simulation includes computing one or more route-specificefficacy metrics of the at least one automated decisioning route of theplurality of distinct automated decisioning routes.

In one embodiment, the one or more efficacy benchmarks comprise aminimum decisioning accuracy value that is calculated based on one ormore accuracy metric values of the incumbent automated decisioningworkflow.

In one embodiment, the evaluating the succeeding automated decisioningworkflow includes comparing the one or more simulation metrics of thesucceeding automated decisioning workflow against the one or morein-production metrics of the incumbent automated decisioning workflowand estimating whether the succeeding automated decisioning workflowmitigates or ameliorates the anomaly based on the comparison, whereinreplacing the incumbent automated decisioning with the succeedingautomated decisioning workflow is based on the estimation of whether thesucceeding automated decisioning workflow mitigates or ameliorates theanomaly.

In one embodiment, a method for adapting an errant automated decisioningworkflow includes reconfiguring digital abuse or digital fraud logicparameters associated with one or more automated decisioning routes ofan automated decisioning workflow in response to identifying ananomalous drift or an anomalous shift in one or more efficacy metrics ofthe automated decisioning workflow, wherein the automated decisioningworkflow comprises a plurality of distinct automated decisioning routesthat, when applied in a digital threat evaluation of data associatedwith a target digital event, automatically compute a decision fordisposing the target digital event based on a probability digital fraudor digital abuse associated with the target digital event; simulating,by one or more computers, a performance of the one or more automateddecisioning routes in a reconfigured state based on inputs of historicaldigital event data input to the incumbent automated decisioning workflowduring a historical period; calculating one or more simulation metricsbased on simulation output data of the simulation of the automateddecisioning workflow; and promoting to an in-production state theautomated decisioning workflow having the one or more automateddecisioning routes in the reconfigured state.

In one embodiment, reconfiguring the one or more automated decisioningroutes of the automated decisioning workflow includes generating newfraud logic parameters that mitigate the anomalous drift or theanomalous shift of the automated decisioning workflow.

In one embodiment, the anomalous drift relates to a gradual change inmetrics associated with decisioning outputs of the one or more automateddecisioning routes over a period exceeding a minimum number of days.

In one embodiment, the anomalous shift relates to an abrupt change inmetrics associated with decisioning outputs of the one or more automateddecisioning routes over an abbreviated period not exceeding a maximumnumber of days or a maximum number of hours.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a schematic representation of a system in accordancewith one or more embodiments of the present application;

FIG. 2 illustrates an example method in accordance with one or moreembodiments of the present application;

FIG. 3 illustrates an example schematic of shadow testing of anautomated decisioning workflow in accordance with one or moreembodiments of the present application; and

FIG. 4 illustrates an example schematic of backtesting of an automateddecisioning workflow in accordance with one or more embodiments of thepresent application.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the presentapplication are not intended to limit the inventions to these preferredembodiments, but rather to enable any person skilled in the art to makeand use these inventions.

Overview

As discussed above, digital threats are abounding and continue to evolveto circumvent existing digital fraud detection technologies. Theevolving nature of digital threats compounded with the great number oftransactions, events, actions, and/or activities (exceeding billions innumber) occurring over the web and/or Internet highlight the manydeficiencies of traditional digital fraud detection and threatmitigation implementations.

The embodiments of the present application, however, provide an advancedtechnology platform that is capable of ingesting billions of digitalevents and/or transactions over the Internet, the web, web applications,mobile applications, and the like and dynamically implement digitalthreat mitigation implementations that are capable of detectingmalicious activities, fraudulent activities, digital abuses and generatedigital threat mitigation recommendations and responses that operate tomitigate and/or eliminate the digital fraud and abuse threats stemmingfrom the malicious or fraudulent activities, as described in U.S. Pat.No. 9,954,879, which is incorporated herein in its entirety by thisreference.

The advanced technology platform of many embodiments of the presentapplication employs a robust ensemble of machine learning models andrelated systems that operate to ingest the great number of digitalactivities performed and events occurring over the web. Accordingly,using these finely tuned and perpetually evolving and tunable machinelearning models, a system implementing the several embodiments of thepresent application can predict a threat level and/or classify a digitalthreat with high accuracy and, in some embodiments, in real-time (e.g.,as the event is occurring or shortly thereafter) compute a digitalthreat score for each event or activity that is received by the system.

The digital threat score may be exposed via a score application programinterface (API) that may function to interact with various endpoints ofthe digital threat mitigation platform. Specifically, the score API mayfunction to interact with one or more computing servers that implementthe ensembles of machine learning models used to predict a likelihood ofdigital fraud and/or digital abuse. The score API may function to returna value (e.g., a number, likelihood or probability, or other criterion)that indicates how likely it is that an actor involved or associatedwith digital events and/or activities is a malicious actor or may beperpetrating cyber fraud or digital abuse (e.g., payment abuse, etc.).Accordingly, the digital threat score calculated by the score API may beused in several manners including to inform digital event dataprocessing decisions (e.g., deny, hold, or approve digital transaction)or to define which of one or more digital threat mitigation protocols orimplementations that should be applied to future digital event dataand/or current the digital events to mitigate or eliminate a digitalthreat associated therewith. Additionally, or alternatively, in one ormore embodiments, digital event data processing decisions may beautomated via automated decisioning workflows, as described in U.S. Pat.No. 9,954,879, which is incorporated herein in its entirety by thisreference and digital event data processing decisions may be performedwith manual input from one or more human analysts or the like. In suchembodiments, decisions output from one or more review queues of the oneor more human analysts may be converted to training data for trainingand improving one or more threat classifying models of the threatmitigation service including, but not limited to, a unified threatmodel.

Workflows Overview

Additionally, the embodiments of the present application further enablethe configuration of new digital threat mitigation automated decisioningworkflows or implementations automatically upon a detection of a new (orevolved) digital threat or semi-automatically with digital threat inputfrom an entity, as described in U.S. Pat. No. 9,954,879, which isincorporated herein in its entirety by this reference. That is, in someinstances, the digital events and activities occurring via the Internetor web (including web and mobile applications) give rise to anunclassified or not fully classified potential digital threat that mayrequire additional or new digital threat mitigation measures that mayeffectively mitigate the fraud risks associated the digital events. Insuch instances, the embodiments of the present application employ adigital threat mitigation engine that is capable of building new digitalthreat mitigation automated workflows or implementations that functionto mitigate or eliminate digital threats posed by the unclassified ornot fully classified potential digital threat. It shall be noted that,in some embodiments, the digital event posing the potential digitalthreat may be a classified digital event (e.g., payment abuse), however,due to the digital risks or other risks of loss, enhanced or new digitalthreat detection automated workflows according to the severalembodiments of the present application may need to be implemented in thecourse of evaluating and processing the digital event.

Using a combination of the digital threat score calculated for a digitalactor and/or digital event and a web user interface, the embodiments ofthe present application enable the definition of new or evolved digitalthreat mitigation automated workflows executed in association with adigital threat mitigation engine. In particular, via the web userinterface, it is possible to identify or define digital events ordigital activities occurring over the web or Internet that may be usedto trigger a digital intervention (e.g., implementation the new digitalthreat mitigation automated workflows) and digital event or digitalactivity processing. The web user interface may similarly be used todefine the specific routines and procedures executed by the digitalthreat mitigation engine once the threat mitigation automated workflowshave been triggered.

The digital threat mitigation engine functions to use, as input, adigital threat score and service provider-defined digital threatmitigation automated workflows to generate one or more digital threatmitigation implementations, such as a digital threat mitigation flow.For instance, the digital threat mitigation engine may function tofurther configure an events API to collect and/or parse events oractivity data from multiple events data sources to identify specificevents data that may trigger the new digital threat mitigationimplementations. In such instance, one aspect of the one or more digitalthreat mitigation implementations may include digitizing defined threatmitigation policy for integration and enhancement of threat mitigationcapabilities of a pre-existing and broader digital threat mitigationservice.

Workflow Testing Overview

Automated decisioning workflows may be used to manage the logic thatdetermines which online users, digital orders, digital content, digitalsessions and/or the like that may be blocked, given friction or review,or allowed to pass without friction.

One or more embodiments of the present application may enable dynamichandling of new automated decisioning routes into an existing automateddecisioning workflow and/or a validation of new digital threatmitigation automated decisioning workflows.

In one or more embodiments, dynamic handling may include edits,adaptations, and/or upgrades to an existing automated decisioningworkflow based on simulations or proposed automated workflow testing. Inone or more embodiments, testing a proposed automated decisioningworkflow may include, but is not limited to, simulating one or morealternate forms of a currently existing/incumbent automated decisioningworkflow and analyzing the results to determine the outcomes that mayhave been achieved if one or more simulated versions of an automateddecisioning workflow were active/live or in production. In one or moreinstances of the dynamic handling and testing/simulation, the digitalevents and activities occurring via the Internet or web (including weband mobile applications) may give rise to usertraffic/events/transactions data that may require proposing entirely newautomated decisioning workflows. In one or more of such examples, it mayno longer be sufficient to test/simulate a modified version of acurrently existing/incumbent version of an automated decisioningworkflow. Thus, some embodiments of the present application furtherenable a creation of an entirely new automated decisioning workflow andperforming simulations and/or workflow testings that may function tovalidate an efficacy of the newly proposed automated decisioningworkflow.

1. System for Machine Learning-Based Identification of Digital Fraudand/or Abuse Detection

As shown in FIG. 1, a system 100 for detecting digital fraud and/ordigital abuse includes one or more digital event data sources 110, a webinterface 120, a digital threat mitigation platform 130, and a serviceprovider system 140.

The system 100 functions to enable a prediction of multiple types ofdigital abuse and/or digital fraud within a single stream of digitalevent data. The system 100 provides web interface 120 that enablessubscribers to and/or customers of a threat mitigation serviceimplementing the system 100 to generate a request for a global digitalthreat score and additionally, make a request for specific digitalthreat scores for varying digital abuse types. After orcontemporaneously with receiving a request from the web interface 120,the system 100 may function to collect digital event data from the oneor more digital event data sources 110. The system 100 using the digitalthreat mitigation platform 130 functions to generate a global digitalthreat score and one or more specific digital threat scores for one ormore digital abuse types that may exist in the collected digital eventdata.

The one or more digital event data sources 110 function as sources ofdigital events data and digital activities data, occurring fully or inpart over the Internet, the web, mobile applications, and the like. Theone or more digital event data sources 110 may include a plurality ofweb servers and/or one or more data repositories associated with aplurality of service providers. Accordingly, the one or more digitalevent data sources 110 may also include the service provider system 140.

The one or more digital event data sources 110 function to captureand/or record any digital activities and/or digital events occurringover the Internet, web, mobile applications (or other digital/Internetplatforms) involving the web servers of the service providers and/orother digital resources (e.g., web pages, web transaction platforms,Internet-accessible data sources, web applications, etc.) of the serviceproviders. The digital events data and digital activities data collectedby the one or more digital event data sources 110 may function as inputdata sources for a machine learning system 132 of the digital threatmitigation platform 130.

The digital threat mitigation platform 130 functions as an engine thatimplement at least a machine learning system 132 and, in someembodiments, together with a warping system 133 to generate a globalthreat score and one or more specific digital threat scores for one ormore digital abuse types. The digital threat mitigation platform 130functions to interact with the web interface 120 to receive instructionsand/or a digital request for predicting likelihoods of digital fraudand/or digital abuse within a provided dataset. The digital threatmitigation engine 130 may be implemented via one or more specificallyconfigured web or private computing servers (or a distributed computingsystem) or any suitable system for implementing system 100 and/or method200.

The machine learning system 132 functions to identify or classifyfeatures of the collected digital events data and digital activity datareceived from the one or more digital event data sources 110. Themachine learning system 132 may be implemented by a plurality ofcomputing servers (e.g., a combination of web servers and privateservers) that implement one or more ensembles of machine learningmodels. The ensemble of machine learning models may include hundredsand/or thousands of machine learning models that work together toclassify features of digital events data and namely, to classify ordetect features that may indicate a possibility of fraud and/or abuse.The machine learning system 132 may additionally utilize the input fromthe one or more digital event data sources 110 and various other datasources (e.g., outputs of system 100, system 100 derived knowledge data,external entity-maintained data, etc.) to continuously improve oraccurately tune weightings associated with features of the one or moreof the machine learning models defining the ensembles.

The warping system 133 of the digital threat mitigation platform 130, insome embodiments, functions to warp a global digital threat scoregenerated by a primary machine learning ensemble to generate one or morespecific digital threat scores for one or more of the plurality ofdigital abuse types. In some embodiments, the warping system 133 mayfunction to warp the primary machine learning ensemble, itself, toproduce a secondary (or derivative) machine learning ensemble thatfunctions to generate specific digital threat scores for the digitalabuse and/or digital fraud types. Additionally, or alternatively, thewarping system 130 may function to implement a companion machinelearning model or a machine learning model that is assistive indetermining whether a specific digital threat score should be generatedfor a subject digital events dataset being evaluated at the primarymachine learning model. Additionally, or alternatively, the warpingsystem 133 may function to implement a plurality of secondary machinelearning models defining a second ensemble that may be used toselectively determine or generate specific digital threat scores.Accordingly, the warping system 133 may be implemented in variousmanners including in various combinations of the embodiments describedabove.

The digital threat mitigation database 134 includes one or more datarepositories that function to store historical digital event data. Thedigital threat mitigation database 134 may be in operable communicationwith one or both of an events API and the machine learning system 132.For instance, the machine learning system 132 when generating globaldigital threat scores and specific digital threat scores for one or morespecific digital abuse types may pull additional data from the digitalthreat mitigation database 134 that may be assistive in generating thedigital threat scores.

The ensembles of machine learning models may employ any suitable machinelearning including one or more of: supervised learning (e.g., usinglogistic regression, using back propagation neural networks, usingrandom forests, decision trees, etc.), unsupervised learning (e.g.,using an Apriori algorithm, using K-means clustering), semi-supervisedlearning, reinforcement learning (e.g., using a Q-learning algorithm,using temporal difference learning), adversarial learning, and any othersuitable learning style. Each module of the plurality can implement anyone or more of: a regression algorithm (e.g., ordinary least squares,logistic regression, stepwise regression, multivariate adaptiveregression splines, locally estimated scatterplot smoothing, etc.), aninstance-based method (e.g., k-nearest neighbor, learning vectorquantization, self-organizing map, etc.), a regularization method (e.g.,ridge regression, least absolute shrinkage and selection operator,elastic net, etc.), a decision tree learning method (e.g.,classification and regression tree, iterative dichotomiser 3, C4.5,chi-squared automatic interaction detection, decision stump, randomforest, multivariate adaptive regression splines, gradient boostingmachines, etc.), a Bayesian method (e.g., naïve Bayes, averagedone-dependence estimators, Bayesian belief network, etc.), a kernelmethod (e.g., a support vector machine, a radial basis function, alinear discriminate analysis, etc.), a clustering method (e.g., k-meansclustering, density-based spatial clustering of applications with noise(DBSCAN), expectation maximization, etc.), a bidirectional encoderrepresentation form transformers (BERT) for masked language model tasksand next sentence prediction tasks and the like, variations of BERT(i.e., ULMFiT, XLM UDify, MT-DNN, SpanBERT, RoBERTa, XLNet, ERNIE,KnowBERT, VideoBERT, ERNIE BERT-wwm, GPT, GPT-2, GPT-3, ELMo,content2Vec, and the like), an associated rule learning algorithm (e.g.,an Apriori algorithm, an Eclat algorithm, etc.), an artificial neuralnetwork model (e.g., a Perceptron method, a back-propagation method, aHopfield network method, a self-organizing map method, a learning vectorquantization method, etc.), a deep learning algorithm (e.g., arestricted Boltzmann machine, a deep belief network method, aconvolution network method, a stacked auto-encoder method, etc.), adimensionality reduction method (e.g., principal component analysis,partial least squares regression, Sammon mapping, multidimensionalscaling, projection pursuit, etc.), an ensemble method (e.g., boosting,bootstrapped aggregation, AdaBoost, stacked generalization, gradientboosting machine method, random forest method, etc.), and any suitableform of machine learning algorithm. Each processing portion of thesystem 100 can additionally or alternatively leverage: a probabilisticmodule, heuristic module, deterministic module, or any other suitablemodule leveraging any other suitable computation method, machinelearning method or combination thereof. However, any suitable machinelearning approach can otherwise be incorporated in the system 100.Further, any suitable model (e.g., machine learning, non-machinelearning, etc.) may be implemented in the various systems and/or methodsdescribed herein.

The service provider 140 functions to provide digital events data to theone or more digital event data processing components of the system 100.Preferably, the service provider 140 provides digital events data to anevents application program interface (API) associated with the digitalthreat mitigation platform 130. The service provider 140 may be anyentity or organization having a digital or online presence that enableusers of the digital resources associated with the service provider'sonline presence to perform transactions, exchanges of data, perform oneor more digital activities, and the like.

The service provider 140 may include one or more web or privatecomputing servers and/or web or private computing devices. Preferably,the service provider 140 includes one or more client devices functioningto operate the web interface 120 to interact with and/or communicationwith the digital threat mitigation engine 130.

The web interface 120 functions to enable a client system or clientdevice to operably interact with the remote digital threat mitigationplatform 130 of the present application. The web interface 120 mayinclude any suitable graphical frontend that can be accessed via a webbrowser using a computing device. The web interface 120 may function toprovide an interface to provide requests to be used as inputs into thedigital threat mitigation platform 130 for generating global digitalthreat scores and additionally, specific digital threat scores for oneor more digital abuse types. Additionally, or alternatively, the web(client) interface 120 may be used to collect manual decisions withrespect to a digital event processing decision, such as hold, deny,accept, additional review, and/or the like. In some embodiments, the webinterface 120 includes an application program interface that is inoperable communication with one or more of the computing servers orcomputing components of the digital threat mitigation platform 130.

The web interface 120 may be used by an entity or service provider tomake any suitable request including requests to generate global digitalthreat scores and specific digital threat scores. In some embodiments,the web interface 120 comprises an application programming interface(API) client and/or a client browser.

Additionally, as shown in FIG. 2-FIG. 6, the systems and methodsdescribed herein may implement the digital threat mitigation platform inaccordance with the one or more embodiments described in the presentapplication as well as in the one or more embodiments described in U.S.patent application Ser. No. 15/653,373, which is incorporated byreference in its entirety.

2. Method for Optimizing Machine Learning-Based Automated DecisioningWorkflows

As shown in FIG. 2, a method 200 for optimizing machine learning-basedautomated decisioning workflows includes sourcing automated decisioningworkflows testing data S205, identifying changes in fraud logic S210,configuring automated decisioning workflow simulation or test parametersS220, implementing simulations and testing of proposed routes inproposed automated decisioning workflows S230, creating statisticalautomated decisioning workflow evaluations S240, and deploying asucceeding automated decisioning workflow S250.

2.05 Identifying Variations in Fraud Logic

Optionally, S205, which includes identifying variations in digital abuseand/or digital fraud logic, may function to compute or identifyinstances in which digital abuse and/or digital fraud variations orchanges in patterns of events or activities of a subscriber mayadversely affect a performance of and/or logic thresholds of one or moreautomated decisioning workflows. In one embodiment, S205 may function torecognize, based on computed variations metrics of a target workflowbased on various stemming from fraud logic of the target workflow, aneed to recalibrate one or more decisioning attributes of automateddecisioning workflows of one or more subscribers.

In one or more embodiments, fraud or workflow logic may refer to a setof instructions (e.g., distinct fraud thresholds mapped to decisionroutes) that may be executed to identify and handle digital events thatmay be fraudulent. In some embodiments, S205 may enable a dynamichandling (change/update/reconfiguring) of fraud logic parameters bysubscribers from time-to-time due to variations in a variety of factorsincluding, but not limited to, fraud patterns, a shift in useractivities pattern (e.g., more mobile users over time), system-generatedthreat score distributions, changes in services or product offerings ofa subscriber, and/or the like.

In some embodiments, a performance of automated decisioning workflowsmay be measured by a plurality of workflow metrics. A relative weightmay be assigned to each workflow metric depending on attributes of asubscriber by a threat mitigation service implementing the method 200.In one or more embodiments, automated decisioning workflow metrics maybe monitored or measured at some predetermined intervals (e.g., at theend of each day, week, month and year etc.) to identify whethervariations may exist in fraud patterns and/or fraud logic typicallyassociated with a target automated decisioning workflow.

In a preferred embodiment, S205 may function to identify and predict aneed for changes in fraud logic before the subscribers/customers orother end users experience a decline in performance level (e.g.,increased frequency of manual review beyond a threshold value, increasedfrequency of fraudulent transaction attempts beyond a threshold value,increased frequency of user complaints beyond a threshold value, etc.)associated with a target automated decisioning workflow. In suchpreferred embodiment, S205 may function to preemptively expose one ormore signals/triggers for modification of one or more automateddecisioning workflow parameters and may alert the subscribers/customersor other end users to implement automated decisioning workflowmodifications through simulation and/or testing techniques, as describedherein, to improve automated decisioning workflow logic by usinghistorical and/or live data in simulations to identify and make one ormore changes in decisioning logic of the automated decisioning workflowbased on overall workflow simulation outcomes.

In one or more preferred embodiments, S205 may function to enablesubscribers/customers or other end users to view, via a web-basedconsole or the like, simulation metrics corresponding to one or moreroutes in one or more proposed automated decisioning workflows over timein automated decisioning workflow reports and may show alerts forchanges/updates in automated decisioning workflow metrics.

One or more of these embodiments may improve efficiency of one or moreautomated decisioning workflows by triggering an implementation of asolution before a reduction in the performance of an automateddecisioning workflow may be detected by subscriber systems orthird-party integrators or the like. In one or more embodiments, S205may function to propose intelligent modifications or updates toautomated decisioning workflows in advance of a threshold level ofdecline in one or more performance parameters of one or more automateddecisioning workflows may be observed. For instance, in one or moreembodiments, S205 may function to identify a potential decline in anefficacy of an automated decisioning workflow based on a detection ofeither a high level of false negatives or a high level of falsepositives based on outcomes of a subject automated decisioning workflow,as described in U.S. patent application Ser. No. 17/083,292, which isincorporated herein in its entirety by this reference. In anotherexample, S205 may function to detect anomalous fraud logic patternsand/or the like based on an evaluation of distributions of the automateddecisions of a subject automated decisioning workflow, as described inU.S. patent application Ser. No. 17/109,639, which is incorporatedherein in its entirety by this reference.

In one or more variants, S205 may function to provide alerts/suggestionsto subscribers/customers about potential changes based on the fraudlogic assessment and related fraud or digital abuse patterns that may bemade to one or more automated decisioning workflows to improve one ormore performance metrics.

2.10 Sourcing Workflow Testing Data

S210, which includes sourcing automated decisioning workflow testingdata, may function to source automated decisioning workflow testing databy collecting a corpus of historical events data, building a corpus ofevents data from live events data, or by building a corpus that includesa subset of historical and/or live event data. In one or moreembodiments, if a workflow testing or simulation includes an incumbentor existing automated decisioning workflow, S210 may function to collecta corpus of historical events data, which may include events datapreviously used as model input into one or machine learning-based threatscoring models. In some embodiments, if a workflow testing or simulationincludes a new or succeeding automated decisioning workflow, S210 mayfunction to build a simulation corpus by sourcing live traffic or eventsdata.

In one or more embodiments, S210 may function to include historical datafrom a predetermined period or time window (e.g., 30-90 days). The timewindow, in such embodiments, may be determined by a subscriber/customerand/or other end user and/or may be dictated by one or more thresholdvalue requirements of a statistical or evaluation algorithm employed togenerate simulation results in a visual form. In such embodiments, thesubscriber/customer may select a data range between current/present timeand/or date and the last possible date/time.

In some embodiments, simulation and testing of one or more decisionroutes in one or more automated decisioning workflows may be performedwith data up to a point of a last edit or modification of one or moreroutes in the one or more automated decisioning workflows.

In one or more embodiments, when live data may be used fortesting/simulation purposes, collection of training data samples mayoccur in parallel with testing. In some of these embodiments, S210 mayfunction with a lag between data ingestion and testing.

2.20 Configuring a Workflow Simulation or Test Parameters

S220, which includes configuring an automated decisioning workflowsimulation or test parameters, may function to receive or set one ormore simulation configurations for a target automated decisioningworkflow for testing one or more new or modified routes of the targetautomated decisioning workflow.

In one or more embodiments, S220 may function to suggest and/ordetermine modifications/alterations in one or more distinct routes of anautomated decisioning workflow. In such embodiments, S220 may functionto include constructing digital threat mitigation automated decisioningworkflows according to the digital threat mitigation policy or automateddecisioning workflow criteria. In such embodiments, the digital threatmitigation policy and/or automated decisioning workflow criteria may beinformed by a plurality of distinct automated workflows deployed acrossthe threat service platform for a plurality of distinct subscribers tothe threat service.

In one or more embodiments, S220 may function to propose changes/updatesin an automated decisioning workflow in the form of addition/deletion ofone or more workflow or decisioning routes, modification of existingroutes, merging existing routes and the like.

In one or more embodiments, S220 may function to employ a single routebuilding experiment or simulation that uses current data: threat scores,fields, and machine learning model features at the time of the automateddecisioning workflow event.

In some embodiments, configuring or defining parameters for one or moreroutes of a new or proposed automated decisioning workflow may includeidentifying one or more threat score threshold values for evaluating acomputed machine learning-based threat score for a given event andidentifying one or more corresponding rules or routes that may triggerthe actions that should be taken automatically by the threat service forhandling the given event if a corresponding threat score threshold issatisfied.

In a first implementation, S220 may function to create a copy of atarget automated decisioning workflow. In such first implementation, oneor more routes in the copy or duplicate of the target automateddecisioning workflow may be modified based on one or more simulationparameters and implemented in an automated decisioning workflowsimulation or automated decisioning workflow testing.

In a second implementation, S220 may function to create a copy of atarget automated decisioning workflow and add one or more routes to thecopy or duplicate of the target automated decisioning workflow in orderto modify, improve or augment the capabilities of currently existingautomated decisioning workflow.

In a third implementation, S220 may function to create an entirely newautomated decisioning workflow without deference to a currently existingor live version of automated decisioning workflow. In one or moreinstances, the new automated decisioning workflow may comprise of adifferent number of routes and/or different threat score thresholdvalue/s for one or more routes compared to the currently existing orlive version of automated decisioning workflow.

In a fourth implementation, S220 may function to isolate a single targetroute in an automated decisioning workflow to allow for testing of thetarget route in isolation or in a vacuum. In this fourth implementation,simulating a single target route for a prospective automated decisioningworkflow may function to simulate a performance of the single targetroute alone and without other routes that may typically precede or beprioritized ahead of the target single route in a full automateddecisioning workflow. One or more of these implementations may be usefulwhen it may take a longer time for user traffic/activity/transactionsdata to reach the target route if a full automated decisioning workflowtesting is employed and/or when data flow to one route may notsignificantly affect the other routes in an automated decisioningworkflow.

2.22 Subscriber-Defined Configurations

S220 includes S222, which may include identifying a proposed automateddecisioning workflow for testing and/or simulations, may function to setor identify parameters for constructing or creating proposed decisionroutes or a proposed automated decisioning workflow based onsubscriber-informed configurations. That is, in some embodiments, S222may function to collect one or more proposed configurations for definingone or more aspects of an automated decisioning workflow which mayinclude modifying one or more existing decisioning or workflow routes ofa target automated decisioning workflow, creating an entirely newautomated decisioning workflow without deference to an existing orincumbent automated decisioning workflow, and/or creating or definingone or more new decisioning routes that may be added to an incumbent orexisting automated decisioning workflow.

In one or more embodiments, S220 may function to enablesubscribers/customers or other end users to propose a new automateddecisioning workflow that may be tested/simulated with live orhistorical user activity/events data alongside a currentlyexisting/incumbent automated decisioning workflow.

In one or more embodiments, S222 may enable subscribers to iterate tocraft a decision route candidate for a target or prospective automateddecisioning workflow. One or more of these embodiments may allowsubscribers/customers or other end users to modify threat scorethreshold values and actions that an automated decisioning workflow maytake if the corresponding threshold threat score value is satisfied. Oneor more of these embodiments may enable the subscribers/customers orother end users to add and/or eliminate one or more routes in one ormore automated decisioning workflows generated for testing orsimulation.

In one or more embodiments, S222 may function to include instances whensubscribers/customers or other end users may identify a need to modifyone or more target routes in one or more target automated decisioningworkflows. In one or more implementations, modifying one or more routesin one or more automated decisioning workflows may have technicalbenefits including, but not limited to improving manual reviewefficiency, mitigating a new type of fraud attack/digital payment abuseand/or the like. In one or more of these implementations, S222 mayfunction to enable subscribers/customers or other end users to proposeone or more variations of the target route/s in one or more newlyproposed/modified/altered versions of the target automated decisioningworkflow for testing/simulation with historical or live data.

In one or more embodiments, S222 may function to enablesubscribers/customers or other end users to propose one or more entirelynew automated decisioning workflows without deference to a currentlyexisting or live version of automated decisioning workflow. In one ormore instances, the new proposed automated decisioning workflow/s maycomprise of a different number of routes and/or different threat scorethreshold value/s for one or more routes compared to the currentlyexisting or live version of automated decisioning workflow.

2.24 System-Defined Configurations Informed by Statistical Analysis

S220 includes S224, which may include threat service-proposed workflowconfiguration changes based on a statistical evaluation of attributes ofautomated decisioning workflow data and/or events data, may function toidentify automated decisioning workflow configurations proposed by athreat service or system implementing the method 200 thatmodifies/alters one or more routes in one or more incumbent automateddecisioning workflows and/or propose new automated decisioning workflowsthat may be tested with live or historical data.

In one or more embodiments, the statistical evaluation may includeidentifying or computing one or more metrics based on trends in fraudpattern, user traffic/events data, changes in decision rates or decisiondistributions of an automated decisioning workflow, and/or the like.

In one or more embodiments, S224 may function to include system-definedconfigurations suggestions for new routes via statistical analysis. Inone or more implementations, S224 may function to propose one or morenew automated decisioning workflows without a deference to an incumbentautomated decisioning workflow.

2.30 Implementing Proposed Workflow Simulations and Testing

S230, which includes implementing simulations and testing of proposedroutes in proposed automated decisioning workflows, may function tosimulate one or more routes in one or more proposed automateddecisioning workflows with historical and/or live traffic or events data(and/or a subset of the data). In one or more embodiments, S230 mayfunction to identify outputs or results of simulations or testing of theone or more proposed automated decisioning workflows thereby informingpotential reconfigurations and/or adaptations to a target or incumbentautomated decisioning workflow of a given subscriber.

2.32 Single Route Backtesting

S230 includes S232, which includes simulating a single route of anautomated decisioning workflow in isolation/vacuum or as a part of afull automated decisioning workflow, may function to simulate aperformance of a target decisioning route apart from existingdecisioning routes of a subject automated decisioning workflow.

In a first implementation, S232 may function to include testing a newproposed route for an automated decisioning workflow. In this firstimplementation, a automated decisioning workflow may include a newproposed decisioning route or a modification of an existing decisioningroute specifically designed to handle new potential digital fraud ordigital abuse patterns (e.g., IP address in a foreign country, occurringwithin an atypical time range, etc.).

In a second implementation, S232 may function to include testing a routewhich is copied from a live version or incumbent automated decisioningworkflow and modified to change parameters, e.g., threat score thresholdvalues and/or the like.

In one or more embodiments, when proposed changes to an automateddecisioning workflow by subscribers or threat service may includechanges or adjustments to one route, e.g., refining a review route todetect more malicious or adverse events, or addition of a route with avery specific purpose e.g., to detect an emerging fraud/abuse or aspecific fraud attack not being detected by a high score block route,S232 may function to test a route using historical data (e.g., 30-90days), wherein the data may include fields, features and the like ofprevious automated decisioning workflows, to generate a summary ofresults that may have been achieved if the route being tested was theonly route running/live/active.

In one or more embodiments, S232 may function to test a target automateddecisioning route in the conditions described above for single routebacktesting but ‘in context of flow’ which means that a proposed routemay be tested such that an output of testing/simulation may not includeautomated decisioning workflow results of routes preceding the targetautomated decisioning route in a prioritized hierarchy of an automateddecisioning workflow.

In one or more embodiments, S232 may function to test a route in theconditions described above for single route backtesting but ‘in place’which may include testing the single route fully integrated with otherworkflow routes which may precede and/or follow the single route in anordering or prioritization of workflow routes within an automateddecisioning workflow. Such embodiments may be employed when historicaldata back to a time of the last edit above the route is used but in oneor more embodiments, S232 may allow a simulation/test with historicaldata back to a maximum possible time with a warning for thesubscriber/customer or other end user that automated decisioningworkflow may have been edited over some time window in the past. In oneor more implementations, this mode may be selected via a user interfaceor initiated/suggested by a system implementing the method 200. In oneor more implementations, when this mode is selected/activated bysubscriber/customer or other end user, the user interface may compriseof toggles, drop down choices, test criteria button and/or the like forselecting and/or activating the mode of simulation.

In one or more embodiments, S232 may function to implement one or moreconstraints or limitations in terms of number of simulations/testsconducted per minute per account or a maximum number of criteriasubscribers/customers or other end users may vary in a test simulation.

In some embodiments, single route backtesting may enable multipletechnical advantages including, but not limited to, avoiding unexpectedchanges in outcome rates (block, review, etc.), ensuring that changes toautomated decisioning workflows result in an improvement in performancemetrics. Therefore, in a preferred embodiment, it may be expected thatthe subscribers/customers or other end users who run single routebacktesting prior to pushing a change to an automated decisioningworkflow may make fewer changes in a subsequent period and have improvedworkflow efficacy metrics.

2.34 Full Workflow Shadow Simulation

S230 includes S234, which includes testing a new proposed automateddecisioning workflow with live traffic data, may function to test a newautomated decisioning workflow or an altered/modified version of anautomated decisioning workflow without affecting a currently existingversion of the automated decisioning workflow. The currently existingversion or incumbent automated decisioning workflow may be referred toherein as live version of the automated decisioning workflow, and thealtered/modified version may be referred to herein as shadow version ofthe automated decisioning workflow, may run in parallel and generatedecisioning outputs that may be measured and used for performancemonitoring, evaluation, and updates or modifications to an automateddecisioning workflow.

In one or more embodiments, a shadow version may be turned on or offduring testing/simulation period so that it may use all live usertraffic/activity data or a subset of traffic/activity data. In suchinstances, subscribers/customers or other end users may determine a timeperiod or conditions to trigger on/off modes of the shadow version ofthe automated decisioning workflow.

In one or more embodiments, S234 may function to generate and/or suggesta modified/altered automated decisioning workflow to create a shadowversion of an automated decisioning workflow, test/simulate the shadowversion of the automated decisioning workflow with live user activitydata and collect results of simulation metrics.

In a first implementation, subscribers/customers or other end users maymake a copy of the live version of the automated decisioning workflow tocreate a draft or tentative workflow for the shadow version of theautomated decisioning workflow, wherein the subscribers/customers orother end users may edit one or more workflow routes of the shadowversion of the automated decisioning workflow.

In a second implementation, subscribers/customers or other end users maymake a copy of the live version of the automated decisioning workflow tocreate a draft for the shadow version of the automated decisioningworkflow, wherein the subscribers/customers or other end users may addor delete one or more workflow routes in the shadow version of theautomated decisioning workflow.

In a third implementation, subscribers/customers or other end users maymake an entirely new version of the automated decisioning workflow tocreate a draft for the shadow version of the automated decisioningworkflow, wherein the subscribers/customers or other end users maycreate one or more new proposed workflow routes in the shadow version ofthe automated decisioning workflow.

In one or more embodiments, S234 may function to allowsubscribers/customers or other end users to make modifications to oradd/delete one or more routes in a shadow version of an automateddecisioning workflow during simulation/testing phase. Stateddifferently, subscribers/customers or other end users may edit a runningor active proposed automated decisioning workflow being tested orsimulated in a shadow mode. In some embodiments, a capability to edit arunning shadow mode may allow subscribers/customers or other end usersto dynamically make modifications to enable an accurate comparison withthe live version of the automated decisioning workflow. For example, ashadow version of an automated decisioning workflow may be created totest one or more routes (e.g., routes 1 and 2 may be tested in a shadowversion of an automated decisioning workflow), after a certain period oftime, the subscriber/customer or other end user may make somemodifications in threshold scores of one or more routes in a liveversion of the automated decisioning workflow (e.g., route 3 may nowblock a transaction if payment abuse threat score threshold value is 86instead of 85), and/or the subscriber/customer or other end user maywant to add or delete one or more routes in the live version of theautomated decisioning workflow. In such instances, S234 may function toenable the subscribers/customers or other end users to make theseadjustments in the shadow version of the automated decisioning workflowso that comparison of metrics may be accurate. In a preferredimplementation, when the subscriber/customer or other end user publishesone or more changes in a live version of an automated decisioningworkflow, they may be prompted or given an option to copy the changes toa shadow version of the automated decision workflow if the system findsone or more shadow versions of the automated decisioning workflow inparallel with the live version of the automated decisioning workflow.

In one or more embodiments, S234 may function to include shadow testingwhen a change may not need to take effect immediately and/orsubscribers/customers may want to see the projected effects beforeapplying a change. Additionally, or alternatively, S234 may function todeal with a new trend where historical data may not exist. In one ormore implementations, S234 may function to include a new subscriber thatmay not have generated enough data for backtesting and/or a new productfrom an existing customer/subscriber that may not have generated enoughdata for backtesting.

In one or more embodiments, S234 may function to modify one or moreworkflow routes in an automated decisioning workflow to create a shadowversion of an automated decisioning workflow and test the shadow versionof the automated decisioning workflow in order to generate simulateddecisioning outcomes and simulation metrics.

In one or more embodiments, a plurality of projected/planned automateddecisioning workflows may be tested in parallel or sequentially andcorresponding simulation metrics may be generated and incorporated instatistical analysis of performance of the plurality of the automateddecisioning workflows.

In some embodiments, a shadow version of an automated decisioningworkflow may have one or more constraints in terms of a simulation ortesting time period (e.g., 15 days or the like). In someimplementations, these constraints may be dictated by cost and utility.For instance, cost may exceed an allocated budget if a test/simulationis run on a shadow version of an automated decisioning workflow thatlasts longer than a certain period of time. In one or moreimplementations, such cost may include, but is not limited to costassociated with using computational resources, storing data generated asa result of simulation/testing and/or the like. Additionally, oralternatively, in one or more implementations, improvement in accuracyof results obtained by running a test with a shadow version of theautomated decisioning workflow may give diminishing results, if the testis run longer than a threshold duration. Alternatively, in someinstances, converting a shadow version of an automated decisioningworkflow to a live version of the automated decisioning workflow withina threshold time period after observing improvement in simulationmetrics of the shadow version of the automated decisioning workflow maybe a technical benefit. In one or more instances, the threshold timeperiod to convert shadow version of an automated decisioning workflow toa live version of the automated decision workflow may be critical if thelive version of the automated decisioning workflow shows decliningperformance metrics continuously over time and the shadow version of theautomated decisioning workflow shows improved simulation metricscompared to the live version of the automated decisioning workflow.

In some embodiments, a shadow version of an automated decisioningworkflow may have one or more constraints in terms of number of shadowautomated decisioning workflows running per subscriber account at atime. In one or more implementations, subscribers/customers or other endusers may be able to control/set/configure the number of shadow versionsof an automated decisioning workflow running per account at a time.

2.36 Full Workflow Backtesting

S230 includes S236, which includes testing a new proposed automateddecisioning workflow with historical data, may function to test/simulateone or more new and/or modified/altered routes of one or more automateddecisioning workflows on a predetermined time window of historical dataand compare simulation metrics with currently existing or live automateddecisioning workflow metrics.

In one or more preferred embodiments, S236 may function to enable asubscriber or other end user to view full automated decisioning workflowor a subset of automated decisioning workflow performance as a snapshotor over time via a web-accessible interface or the like.

In one or more embodiments, subscribers/customers may identify a need toedit one or more target routes in an automated decisioning workflow,S236 may function to test potential variations of the target route/susing historical data to make an informed choice.

In one or more embodiments, S236 may function to includedata/information of timings of changes/updates e.g., when automateddecisioning workflows were edited, paused, or changed from run always torun on request.

In a first implementation, subscribers/customers or other end users maymake a copy of a currently existing/incumbent version of an automateddecisioning workflow to create a draft for a proposed version of theautomated decisioning workflow, wherein the subscribers/customers orother end users may edit one or more routes of the proposed version ofthe automated decisioning workflow in order to simulate it withhistorical data of a predetermined time window.

In a second implementation, subscribers/customers or other end users maymake a copy of a currently existing/incumbent version of an automateddecisioning workflow to create a draft for a proposed version of theautomated decisioning workflow, wherein the subscribers/customers orother end users may add or delete one or more routes in the proposedversion of the automated decisioning workflow in order to simulate itwith historical data of a predetermined time window.

In a third implementation, subscribers/customers or other end users maypropose an entirely new version of an automated decisioning workflow tocreate a draft for the proposed version of the automated decisioningworkflow, wherein the subscribers/customers or other end users maycreate one or more new proposed routes in the proposed version of theautomated decisioning workflow in order to simulate it with historicaldata of a predetermined time window.

2.38 A/B Testing

S230 includes S238, which includes scenario-based testing of a proposedautomated decisioning workflow with data allocated partially to eachscenario, may function to identify results of proposed changes in aproposed new/modified automated decisioning workflow, which mayintelligently support informed decisions for changes to or a creation ofan automated decisioning workflow based on what-if scenarios and causeand effect analysis.

In one or more embodiments, S238 may function to apply changes to one ormore routes of an automated decisioning workflow, such that the changesmay be applicable to a small percentage ofevents/activities/transactions.

In one or more embodiments, S238 may function to test two differentautomated decisioning workflows and allocate 50% of user activity/eventsdata to new automated decisioning workflow to determine one or more longterm outcomes.

In one or more embodiments, A/B test may depend on seasons and marketingexpenditure. For example, August vacation in Europe may result in fewertransactions/events/user traffic data from that region. Another examplemay be that marketing efforts for sales in December before Christmas mayresult in a higher number of transactions than usual and hence generatemore user traffic/events data.

In one or more variants, S238 may function to include A/B/C etc. testingto test new fraud logic strategies on a plurality of subsets of eventsdata and observe real results. In one or more of such instances,projected results may not be useful and/or accurate and hencetesting/simulating with live events/activity data (or a subset of liveevents/activity data) may help improve performance of automateddecisioning workflows.

2.40 Creating Statistical Workflow Evaluations

S240, which includes creating statistical automated decisioning workflowevaluations, may function to compute one or more statistical metrics ofa simulation and create corresponding graphical comparisons of aproposed automated decisioning workflow and incumbent automateddecisioning workflow.

In one or more embodiments, S240 may function to enablesubscribers/customers or other end users to detect trends in performanceof one or more routes of one or more automated decisioning workflowsthrough automated decisioning workflow metrics including, but notlimited to a block rate, a false positive rate, a false negative rate,an acceptance rate, a review rate, a fraud rate, and/or the like. In oneor more implementations, this support subscribers/customers or other endusers in addressing potential issues when statistical analysis shows oneor more alarming or adverse trends.

In some embodiments, simulation and/or testing of one or more routes inone or more automated decisioning workflows may be performed with dataup to a point of the last edit of one or more routes in one or moreautomated decisioning workflows, in such embodiments, S240 may functionto enable visualization of a comparison of metrics. For instance, S240may function to create and provide (e.g., display) two sets of barswithin a bar graph or the like to denote metrics for tested/proposedautomated decisioning workflow vs currently existing/incumbent automateddecisioning workflow metrics.

In some embodiments of the present application, when single routebacktesting is employed, S240 may function to augment the visual datarepresentation results with a disclaimer/help text to informsubscribers/customers or other end users that for testing/simulating aroute of an automated decisioning workflow in vacuum and/or with databeyond the last edit date, simulation metrics computed or generated bythe system may not be an exact reflection of results that may have beenproduced by integrating the tested route with rest of the automateddecisioning workflow but rather show results that may have been producedif the tested route had been at the top of the automated decisioningworkflow and captured all possible users/orders and/or the like in itscapacity.

In some embodiments, when full workflow shadow testing is employed, S240may function to store results of simulation metrics for reportingpurposes. In one or more implementations, such results may be stored fora certain time period. In a preferred implementation, this time periodmay be at least 120 days or the like. It shall be noted that anysuitable time period for storing simulation metrics may be employed.Additionally, or alternatively, in a preferred implementation,simulation metrics may be stored in a way that allows for displaying inWorkflow Metrics and Workflow Route Metrics. In one or moreimplementations, data of the results of the simulation metrics may befetched, via an API or the like, for reports under a certain thresholdtime period. In one or more preferred implementations, this thresholdtime period may be 60 seconds or the like.

In some embodiments, when full workflow shadow testing is employed, S240may function to enable subscriber/customer or other end user to viewperformance of one or more automated decisioning workflows in terms ofsimulation metrics over time and compare automated decisioning workflowmetrics of a live version of an automated decisioning workflow andsimulation metrics of one or more shadow versions of the automateddecisioning workflow simultaneously (e.g., on the same graphicalrepresentation and/or chart and/or on the same screen and/or the like).

In one or more preferred embodiments, S240 may function to provide auser interface where subscribers/customers may identify users,transactions, events, activities and/or the like that are targetedtowards one or more distinct routes in one or more distinct automateddecisioning workflows. One or more of these embodiments may enable thesubscribers/customers to visualize the differences between an incumbentautomated decisioning workflow and a succeeding/proposed automateddecisioning workflow. For example, in one or more such instances, S240may function to build a review queue of the users that may have beenblocked by one or more proposed version of an automated decisioningworkflow in a simulation/test but may not have been blocked in alive/currently existing/incumbent version of the automated decisioningworkflow.

In one or more embodiments, S240 may function to enablesubscribers/customers to view workflow metrics or simulation metricsvolume over time by one or more outcome types in a graphicalrepresentation over a certain period of time (hour/day/week/month). Inone or more implementations, various types of outcomes of simulationmetrics may include but are not limited to auto block rates/amount, autowatch rates/amount, auto allow rates/amount, manual block rates/amount,manual watch rates/amount, manual allow rates/amount, review queuetimeout rates/amount, verification rates/amount, no action rates/amount,notification rates/amount, total volume of user activity/traffic dataand/or the like.

In one or more embodiments, S240 may function to enablesubscribers/customers to view cost (in dollars and/or other currencies)associated with each type of fraudulent activity and/or payment abusee.g., chargeback rates, customer insult cost etc. In a preferredimplementation, S240 may function to provide a breakdown of cost withrespect to routes in an automated decisioning workflow, review subtypeand/or the like.

In one or more embodiments, S240 may function to enablesubscribers/customers to compare two or more automated decisioningworkflows that may have run over the same time period, on the sametriggers, using the same simulation metrics, and using the samegraphical representation. In one or more implementations, suchcomparisons may include but are not limited to backtest vs. live,backtest vs. backtest, shadow vs. live, shadow vs. shadow, shadow vs.backtest and/or the like.

2.5 Deploying a Succeeding Automated Decisioning Workflow

S250, which includes deploying a succeeding automated decisioningworkflow, may function to convert a proposed or tested automateddecisioning workflow to a live automated decisioning workflow based on astatistical analysis of the simulations or validations of (i) theproposed modifications of an automated decisioning workflow or (ii) avalidation of routes of a new automated decisioning workflow.

In one or more embodiments, S250 may function to include implementing aproposed digital threat mitigation based automated decisioning workflowin parallel with an existing/incumbent digital threat mitigation schemeand/or an existing/incumbent automated decisioning workflow in order tocompare and measure one or more performance or simulation metrics of theautomated decisioning workflows and to make an informed decision onwhether to replace the existing/incumbent automated decisioning workflowwith the newly generated/constructed/proposed automated decisioningworkflow and/or to augment the existing automated decisioning workflowwith one or more routes of the newly proposed automated decisioningworkflow.

In one or more embodiments, S250 may function to enable asubscriber/user to replace a live/current or existing/incumbent versionof automated decisioning workflow with a shadow/backtested version ofthe automated decisioning workflow.

In one or more embodiments, S250 may function to compare one or moresimulation metrics of one or more proposed routes in a new or modifiedautomated decisioning workflow against one or more automated decisioningworkflow metrics of one or more routes of a currently existing/incumbentautomated decisioning workflow. In such embodiments, if the proposednew/modified automated decisioning workflow shows improved simulationmetrics (e.g., less chargeback cost and/or the like), S250 may thenfunction to update the currently existing/incumbent automateddecisioning workflow and replace it with the proposed/simulated/testedautomated decisioning workflow. Alternatively, in some otherembodiments, if the tested automated decisioning workflow showssimulation metrics indicating a decline in performance compared to theautomated decisioning workflow metrics of a currently existing orincumbent version of an automated decisioning workflow, S250 may thenfunction to ignore the proposed changes and keep the currentlyexisting/incumbent automated decisioning workflow in its original form.

Embodiments of the system and/or method can include every combinationand permutation of the various system components and the various methodprocesses, wherein one or more instances of the method and/or processesdescribed herein can be performed asynchronously (e.g., sequentially),concurrently (e.g., in parallel), or in any other suitable order byand/or using one or more instances of the systems, elements, and/orentities described herein.

The system and methods of the preferred embodiment and variationsthereof can be embodied and/or implemented at least in part as a machineconfigured to receive a computer-readable medium storingcomputer-readable instructions. The instructions are preferably executedby computer-executable components preferably integrated with the systemand one or more portions of the processors and/or the controllers. Thecomputer-readable medium can be stored on any suitable computer-readablemedia such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD orDVD), hard drives, floppy drives, or any suitable device. Thecomputer-executable component is preferably a general or applicationspecific processor, but any suitable dedicated hardware orhardware/firmware combination device can alternatively or additionallyexecute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

We claim:
 1. A method for adapting an errant automated decisioningworkflow for improving digital fraud or digital abuse mitigation, themethod comprising: generating a succeeding automated decisioningworkflow by reconfiguring an incumbent automated decisioning workflowbased on detecting an anomaly in automated decisioning outputs of theincumbent automated decisioning workflow, wherein the incumbentautomated decisioning workflow comprises a plurality of distinctautomated decisioning routes that, when applied in a digital threatevaluation of data associated with a target digital event, automaticallycompute a decision for disposing the target digital event based on aprobability digital fraud or digital abuse associated with the targetdigital event, and wherein generating the succeeding automateddecisioning workflow includes: tuning at least one automated decisioningroute of the plurality of distinct decisioning routes of the incumbentautomated decisioning workflow based on one or more in-productionmetrics of the automated decisioning outputs of the incumbent automateddecisioning workflow; evaluating the succeeding automated decisioningworkflow, wherein the evaluating includes implementing a routesimulation, by one or more computers, of the succeeding automateddecisioning workflow that simulates a performance of the at least oneautomated decisioning route of the plurality of distinct automateddecisioning routes; computing one or more simulation metrics based onsimulation output data of the route simulation of the succeedingautomated decisioning workflow; and replacing the incumbent automateddecisioning with the succeeding automated decisioning workflow if theone or more simulation metrics satisfy or exceed one or more efficacybenchmarks.
 2. The method according to claim 1, wherein the tuning theat least one automated decisioning route includes: estimating arectifying edit that amends one or more decisioning criteria of the atleast one automated decisioning route based on attributes associatedwith the anomaly in the automated decisioning outputs of the incumbentautomated decisioning workflow.
 3. The method according to claim 2,wherein: one of the one or more decisioning criteria of the at least oneautomated decisioning route comprises a machine learning-based scorethreshold, and the rectifying edit includes adapting the machinelearning-based score threshold by increasing or by decreasing a machinelearning score value or a machine learning score range of valuesassociated with the machine learning-based score threshold.
 4. Themethod according to claim 2, wherein: one of the one or more decisioningcriteria of the at least one automated decisioning route comprises oneor more adverse feature conditions, and the rectifying edit includesadapting the one or more adverse feature conditions by deleting one ofthe one or more adverse feature conditions or by augmenting the one ormore adverse feature conditions with a new adverse feature condition. 5.The method according to claim 2, wherein: one of the one or moredecisioning criteria of the at least one automated decisioning routecomprises a machine learning-based score threshold defined by either amachine learning score value or a machine learning score range ofvalues, and the rectifying edit includes adapting the at least oneautomated decisioning route to include one or more new adverse featureconditions.
 6. The method according to claim 1, wherein: each of theplurality of distinct automated decisioning routes is arranged in apredetermined order within a sequence of the plurality of distinctautomated decisioning routes, and the tuning the at least one automateddecisioning route includes re-positioning the at least one automateddecisioning route to a new position within the sequence of the pluralityof distinct automated decisioning routes.
 7. The method according toclaim 1, wherein: each of the plurality of distinct automateddecisioning routes is arranged in a predetermined order within asequence of the plurality of distinct automated decisioning routes, andimplementing the route simulation of the succeeding automateddecisioning workflow includes: simulating the performance of the atleast one automated decisioning route independently and outside of thesequence of the plurality of distinct automated decisioning routes. 8.The method according to claim 1, wherein: each of the plurality ofdistinct automated decisioning routes is arranged in a predeterminedorder within a sequence of the plurality of distinct automateddecisioning routes, and implementing the route simulation of thesucceeding automated decisioning workflow includes: simulating theperformance of the at least one automated decisioning route in-place andwithin the sequence of the plurality of distinct automated decisioningroutes.
 9. The method according to claim 1, wherein implementing theroute simulation of the succeeding automated decisioning workflowincludes: defining a simulation corpus of data comprising historicaldigital event data input to the incumbent automated decisioning workflowduring a historical period; and simulating the performance of the atleast one automated decisioning route based on inputs of the simulationcorpus.
 10. The method according to claim 1, wherein implementing theroute simulation of the succeeding automated decisioning workflowincludes: setting the route simulation to a shadow mode, wherein duringthe shadow mode a copy of live digital event data passing through theincumbent automated decisioning workflow is additionally passed throughthe succeeding automated decisioning workflow, wherein in the shadowmode automated decisioning outputs of the succeeding automateddecisioning workflow are not exposed via an application programminginterface.
 11. The method according to claim 10, wherein during animplementation of the shadow mode, the route simulation is performed forall automated decisioning routes of the plurality of distinct automateddecisioning routes of the succeeding automated decisioning workflow. 12.The method according to claim 1, wherein the computing the one or moreefficacy metrics based on the route simulation includes computing one ormore route-specific efficacy metrics of the at least one automateddecisioning route of the plurality of distinct automated decisioningroutes.
 13. The method according to claim 1, wherein the one or moreefficacy benchmarks comprise a minimum decisioning accuracy value thatis calculated based on one or more accuracy metric values of theincumbent automated decisioning workflow.
 14. The method according toclaim 1, wherein the evaluating the succeeding automated decisioningworkflow includes comparing the one or more simulation metrics of thesucceeding automated decisioning workflow against the one or morein-production metrics of the incumbent automated decisioning workflow,the method further comprises: estimating whether the succeedingautomated decisioning workflow mitigates or ameliorates the anomalybased on the comparison, wherein replacing the incumbent automateddecisioning with the succeeding automated decisioning workflow is basedon the estimation of whether the succeeding automated decisioningworkflow mitigates or ameliorates the anomaly.
 15. A method for adaptingan errant automated decisioning workflow, the method comprising:reconfiguring digital abuse or digital fraud logic parameters associatedwith one or more automated decisioning routes of an automateddecisioning workflow in response to identifying an anomalous drift or ananomalous shift in one or more efficacy metrics of the automateddecisioning workflow, wherein the automated decisioning workflowcomprises a plurality of distinct automated decisioning routes that,when applied in a digital threat evaluation of data associated with atarget digital event, automatically compute a decision for disposing thetarget digital event based on a probability digital fraud or digitalabuse associated with the target digital event; simulating, by one ormore computers, a performance of the one or more automated decisioningroutes in a reconfigured state based on inputs of historical digitalevent data input to the incumbent automated decisioning workflow duringa historical period; calculating one or more simulation metrics based onsimulation output data of the simulation of the automated decisioningworkflow; and promoting to an in-production state the automateddecisioning workflow having the one or more automated decisioning routesin the reconfigured state.
 16. The method according to claim 15, whereinreconfiguring the one or more automated decisioning routes of theautomated decisioning workflow includes generating new fraud logicparameters that mitigate the anomalous drift or the anomalous shift ofthe automated decisioning workflow.
 17. The method according to claim16, wherein the anomalous drift relates to a gradual change in metricsassociated with decisioning outputs of the one or more automateddecisioning routes over a period exceeding a minimum number of days. 18.The method according to claim 16, wherein the anomalous shift relates toan abrupt change in metrics associated with decisioning outputs of theone or more automated decisioning routes over an abbreviated period notexceeding a maximum number of days or a maximum number of hours.